How can we sample some of the variables in a pyll configuration space, while assigning values to the others?

Let's look at a simple example involving 2 variables 'a' and 'b'. The 'a' variable controls whether our space returns -1 or some random number, 'b'.

If we just run optimization normally, then we'll find that 'a' should be 0 (the index of the choice that gives the lowest return value.



In [9]:

    
from hyperopt import hp, fmin, rand
space = hp.choice('a', [-1, hp.uniform('b', 0, 1)])
best = fmin(fn=lambda x: x, space=space, algo=rand.suggest, max_evals=100)
print best









    



{'a': 0}

But what if someone else already set up the space, and we just run the search over the other part of the space, which corresponds to the uniform draw?

The easiest way to do this is probably to clone the search space, while making some substitutions while we're at it. We can just make a new search space in which 'a' is no longer a hyperparameter.



In [10]:

    
# put the configuration space in a local var
# so that we can work on it.
print space









    



0 switch
1   hyperopt_param
2     Literal{a}
3     randint
4       Literal{2}
5      rng =
6       Literal{<mtrand.RandomState object at 0x7f3bdf6f6630>}
7   Literal{-1}
8   float
9     hyperopt_param
10       Literal{b}
11       uniform
12         Literal{0}
13         Literal{1}
14        rng =
15         Literal{<mtrand.RandomState object at 0x7f3bdf6f6630>}  [line:6]

The transformation we want to make on the search space is to replace the randint with a constant value of 1, corresponding to always choosing hyperparameter a to be the second element of the list of choices.

Now, if you don't have access to the code that generated a search space, then you'll have to go digging around for the node you need to replace. There are two approaches you can use to do this: navigation and search.



In [13]:

    
from hyperopt import pyll

# The "navigation" approach to finding an internal
# search space node:
randint_node_nav = space.pos_args[0].pos_args[1]
print "by navigation:"
print randint_node_nav

# The "search" approach to finding an internal
# search space node:
randint_nodes = [node for node in pyll.dfs(space) if node.name == 'randint']
randint_node_srch, = randint_nodes
print "by search:"
print randint_node_srch

assert randint_node_nav == randint_node_srch









    



by navigation:
0 randint
1   Literal{2}
2  rng =
3   Literal{<mtrand.RandomState object at 0x7f3bdf6f6630>}
by search:
0 randint
1   Literal{2}
2  rng =
3   Literal{<mtrand.RandomState object at 0x7f3bdf6f6630>}



In [16]:

    
space_with_fixed_a = pyll.clone(space, memo={randint_node_nav: pyll.as_apply(1)})
print space_with_fixed_a









    



0 switch
1   hyperopt_param
2     Literal{a}
3     Literal{1}
4   Literal{-1}
5   float
6     hyperopt_param
7       Literal{b}
8       uniform
9         Literal{0}
10         Literal{1}
11        rng =
12         Literal{<mtrand.RandomState object at 0x7f3bdf6f6630>}

Now, having cloned the space with a new term for the randint, we can search the new space. I wasn't sure if this would work because I haven't really tested the use of hyperopt_params that wrap around non-random nodes (here we replaced the randint with a constant) but it works for random search:



In [19]:

    
best = fmin(fn=lambda x: x, space=space_with_fixed_a, algo=rand.suggest, max_evals=100)
print best









    



{'a': 1, 'b': 0.013493404710812285}

Yep, sure enough: The TPE implementation is broken by a hyperparameter that turns out to be a constant. At implementation time, that was not part of the plan.



In [20]:

    
from hyperopt import tpe
best = fmin(fn=lambda x: x, space=space_with_fixed_a, algo=tpe.suggest, max_evals=100)
print best









    



---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-20-dbeeaa873ed3> in <module>()
      1 from hyperopt import tpe
----> 2 best = fmin(fn=lambda x: x, space=space_with_fixed_a, algo=tpe.suggest, max_evals=100)
      3 print best

/home/bergstra/.VENV/eccv12/src/hyperopt/hyperopt/fmin.pyc in fmin(fn, space, algo, max_evals, trials, rseed)
    326 
    327     rval = FMinIter(algo, domain, trials, max_evals=max_evals)
--> 328     rval.exhaust()
    329     return trials.argmin
    330 

/home/bergstra/.VENV/eccv12/src/hyperopt/hyperopt/fmin.pyc in exhaust(self)
    289     def exhaust(self):
    290         n_done = len(self.trials)
--> 291         self.run(self.max_evals - n_done, block_until_done=self.async)
    292         self.trials.refresh()
    293         return self

/home/bergstra/.VENV/eccv12/src/hyperopt/hyperopt/fmin.pyc in run(self, N, block_until_done)
    244                         print 'trial %i %s %s' % (d['tid'], d['state'],
    245                             d['result'].get('status'))
--> 246                 new_trials = algo(new_ids, self.domain, trials)
    247                 if new_trials is base.StopExperiment:
    248                     stopped = True

/home/bergstra/.VENV/eccv12/src/hyperopt/hyperopt/tpe.pyc in suggest(new_ids, domain, trials, seed, prior_weight, n_startup_jobs, n_EI_candidates, gamma, linear_forgetting)
    797     t0 = time.time()
    798     (s_prior_weight, observed, observed_loss, specs, opt_idxs, opt_vals) \
--> 799             = tpe_transform(domain, prior_weight, gamma)
    800     tt = time.time() - t0
    801     logger.info('tpe_transform took %f seconds' % tt)

/home/bergstra/.VENV/eccv12/src/hyperopt/hyperopt/tpe.pyc in tpe_transform(domain, prior_weight, gamma)
    772             observed_loss['vals'],
    773             pyll.Literal(gamma),
--> 774             s_prior_weight
    775             )
    776 

/home/bergstra/.VENV/eccv12/src/hyperopt/hyperopt/tpe.pyc in build_posterior(specs, prior_idxs, prior_vals, obs_idxs, obs_vals, oloss_idxs, oloss_vals, oloss_gamma, prior_weight)
    654                 obs_below, obs_above = obs_memo[node]
    655                 aa = [memo[a] for a in node.pos_args]
--> 656                 fn = adaptive_parzen_samplers[node.name]
    657                 b_args = [obs_below, prior_weight] + aa
    658                 named_args = [[kw, memo[arg]]

KeyError: 'asarray'

The TPE algorithm works if we make a different replacement in the graph. If we replace the entire "hyperopt_param" node corresponding to hyperparameter "a", then it works fine.



In [21]:

    
space_with_no_a = pyll.clone(space, memo={space.pos_args[0]: pyll.as_apply(1)})
best = fmin(fn=lambda x: x, space=space_with_no_a, algo=tpe.suggest, max_evals=100)
print best









    



{'b': 0.000269455723739237}